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Abstract 

Effective  acquisition  programs,  in  terms  of  cost  and  capability  outcomes,  are  increasingly 
important  in  today's  cost-constrained  environments.  Thus,  it  is  important  to  have  effective 
decision  support  for  acquisition  policy  and  process  design.  This  paper  discusses  a  simulation- 
based  approach  for  decision  support  that  facilitates  analysis  of  the  effect  of  system  and 
acquisition  enterprise  characteristics  on  acquisition  outcomes  for  different  policy  and  process 
alternatives  (e.g.,  traditional  vs.  evolutionary).  The  particular  characteristics  studied  are  system 
modularity  and  production  quantity,  plus  enterprise  architecture  and  risk  characteristics  (i.e. , 
mission  risk).  The  modeling  approach  and  results  to  date  are  presented. 

1.  Introduction 

With  the  continued  advent  of  new  threats  on  the  one  hand,  and  likely  constraints  on  the 
ability  of  the  government  to  fund  new  systems  on  the  other,  effective  military  acquisition 
programs  are  increasingly  important.  New  threats  currently  derive  from  asymmetric  and 
regional  sources  such  as  terrorism,  insurgencies  and  cyber-warfare.  These  new  threats  call  for 
new  types  of  systems.  However,  the  defense  acquisition  enterprise  operates  in  an  increasingly 
cost-constrained  environment.  In  recent  years,  acquisition  cost  overruns  have  been  highlighted 
by  the  GAO  and  have  provoked  concern  from  government  funding  sources.  In  addition,  short¬ 
term  war  expenditures  have  used,  and  continue  to  use,  funds  that  otherwise  might  have  been 
used  for  the  acquisition  of  new  systems,  and  long-term  government  entitlement  commitments 
may  constrain  future  funding  for  new  systems.  Finally,  sustainment  cost  is  becoming  an 
increasingly  significant  area  of  concern. 

This,  of  course,  is  not  a  new  observation  since  the  past  forty  years  have  seen  numerous 
attempts  at  reforming  the  acquisition  enterprise.  One  of  the  most  important  reforms  is  the 
concept  of  evolutionary  acquisition,  in  which  systems  are  acquired  in  smaller  increments  of 
capability  and  then  evolved  after  initial  deployment  with  capability  upgrades.  The  theory  is  that 
evolutionary  acquisition  enables  shorter  cycles  for  acquisition,  allowing  new  capabilities  to  be 
deployed  more  quickly  to  warfighters  in  the  field  at  less  cost,  as  opposed  to  traditional 
acquisition  approaches  that  rely  on  long  development  cycles  (Johnson  &  Johnson,  2002). 

Despite  evolutionary  acquisition's  status  as  official  policy,  though,  the  Department  of 
Defense  seems  to  have  had  limited  success  in  its  implementation  (Lorell,  Lorrell,  &  Younossi, 
2006).  Our  previous  work  has  demonstrated  that  evolutionary  acquisition  can,  in  fact,  result  in 
quicker  deployment  of  increased  capability  but  that  more  frequent  cycles  incur  additional 
overhead  that  may  increase  overall  costs  (Pennock  &  Rouse,  2008).  By  expanding  on  these 
results,  this  paper  seeks  to  study  the  effect  of  system  and  enterprise  features  on  the 
performance  of  acquisition  policies.  In  particular,  the  immediate  focus  is  on  the  effect  of  system 
modularity  on  acquisition  lifecycle  performance,  where  performance  is  considered  as  (i)  the  time 
taken  to  deploy  new  capabilities  in  the  field,  (ii)  the  availability  of  systems  in  the  field  once 
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deployed,  (iii)  and  the  lifecycle  cost  associated  with  acquisition  and  sustainment.  The  notion  of 
modularity  has  potential  synergy  with  evolutionary  acquisition — in  terms  of  enabling  capability 
upgrades  to  be  integrated  into  existing  platforms — due  to  the  presence  of  a  modular  system 
architecture. 

This  paper  discusses  a  simulation-based  approach  that  provides  decision  support  for  the 
design  of  acquisition  policies  and  processes  over  the  acquisition  lifecycle  so  that  issues  such  as 
the  effect  of  system  modularity  can  be  addressed.  The  remainder  of  the  paper  is  organized  as 
follows.  Section  2  reviews  the  literature  on  system  modularity  in  product  design  and  acquisition 
processes.  Section  3  describes  the  simulation  model  used  in  this  research.  Sections  4  and  5 
discuss  an  initial  experiment  and  its  results,  demonstrating  the  effect  of  modularity  on  costs  and 
availability.  Then,  Section  6  concludes  with  a  description  of  future  research  intentions. 

2.  Literature  Review 

Modularity  is  typically  conceptualized  as  a  matrix  of  relationships  between  different 
system  modules  or  components,  where  the  relationship  may  mean  that  two  modules  or 
components  are  connected  or  that  changes  to  one  impact  the  other.  Here,  we  adopt  the  latter 
as  the  meaning.  For  instance,  a  laptop  computer  is  typically  considered  less  modular  than  a 
desktop  since  many  components  of  a  desktop  are  designed  to  be  assembled  and  replaced  by 
the  user  without  changes  to  other  components  (Holtta-Otto  &  de  Week,  2007).  The  modular 
architecture  of  a  system  often  is  considered  to  consist  of  a  set  of  modules  or  components  and 
an  infrastructure,  which  connects  components  or  otherwise  provides  a  platform  for  the  system. 
Here,  we  adopt  the  terminology  that  a  simple  system  is  composed  of  components  and  that  a 
complex  system  is  composed  of  modules,  which  are,  in  turn,  composed  of  components.  In  this 
type  of  complex  system,  a  module  typically  has  strong  relationships  among  its  constituent 
components. 

Assume  that  a  value  of  1  means  that  two  components  are  strongly  related,  that  a  value 
of  0  means  that  they  are  not  related,  and  that  a  value  in  between  represents  the  probability  that 
they  are  related  over  a  set  of  circumstances.  Figure  1 ,  then,  illustrates  the  concept  of 
modularity  for  small  systems  represented  by  matrices.  It  should  be  noted  in  the  figure  that  the 
matrix  entry  represents  the  degree  to  which  a  change  in  component  /  affects  component  j. 
Also,  the  matrix  representation  is  not  standardized  in  the  literature.  For  instance,  other  efforts 
reverse  the  role  of  the  rows  and  columns  (e.g.,  Baldwin  &  Clark,  2000).  It  is  assumed  that 
entries  along  the  diagonal  are  all  1;  however,  they  are  not  relevant  to  the  model.  In  Figure  1e, 
then,  component  1  is  the  infrastructure,  and  the  example  shows  that  a  change  to  it  impacts  all 
components.  In  Figure  If,  there  are  two  modules,  each  composed  of  two  components. 


DEFENSE  ACQUISITION  IN  TRANSITION 


-95- 


a  I.  Completely  modular 


b  i  W  eak  connections 


c)  Few  connection* 


1 

0 

0 

0 

1 

o 

0 

0 

1 

1 

0 

1 

0 

1 

0 

0 

0 

1 

1 

1 . 

A 

*■& 

1 

■A 

1 . 

1 

d)  Completely  uon-modulai 


1 

1 

1 

1 

1 

1 

1 

1 

1 

e)  With  infrastructure 


1 

1 

1 

0 

1 

0 

0 

ft 

1 

f).  With  modules 


1 

1 

ft 

0 

1 

1 

0 

0 

0 

o 

1 

1 

(1 

0 

1 

1 

Figure  1.  Modularity  Representations 

The  concept  of  modularity  in  system  design  has  been  researched  fairly  extensively  over 
the  last  twenty  years.  Much  of  this  literature  applies  to  commercial  product  design  rather  than 
military  system  design.  In  this  discussion,  the  terms  system  and  product  will  be  used 
interchangeably.  Ulrich  and  Tung  (1991)  offer  one  of  the  first  definitions  of  modularity,  focusing 
on  (i)  similarity  between  the  physical  and  functional  architectures  of  a  product  or  system  and  (ii) 
minimization  of  interactions  between  physical  components.  While  function  is  one  focus  of 
modularity  research,  another  focus  is  on  the  system  lifecycle — for  instance,  modularity  to 
facilitate  component  disassembly,  recycling  or  reuse  (Gershenson,  Prasad  &  Allamneni,  1999). 
The  lifecycle  focus  provides  a  framework  for  discussing  how  modularity  affects  cost  during  the 
different  phases  of  acquisition. 

In  design,  there  is  considerable  literature  on  how  to  format  for  modularity.  The  research 
literature,  for  the  most  part,  does  not  concentrate  directly  on  cost,  though.  Baldwin  and  Clark 
(2000)  discuss  three  stages  of  cost  with  respect  to  designing  for  modularity:  (i)  establishing 
design  rules,  (ii)  establishing  design  parameters,  and  (iii)  testing  and  fixes.  Design  rules  provide 
constraints  within  which  modules  (or  components)  must  operate.  As  the  number  of  modules 
increases,  the  cost  of  establishing  design  rules  also  increases,  although  no  specific  relationship 
is  identified  (Baldwin  &  Clark,  2000).  Establishing  design  rules  is  considered  a  one-time 
expenditure,  since  they  are  believed  to  remain  in  effect  for  a  long  time.  Design  parameters 
must  be  established  each  time  a  module  is  designed.  The  cost  increases  with  product 
complexity  and  is  applied  for  each  redesign.  Costs  for  testing  and  fixes  start  high  but  decrease 
over  time  as  personnel  gain  expertise  with  the  particular  product  or  system. 

Holtta  and  Otto  (2005)  support  the  general  relationship  described  for  Baldwin  and 
Clark’s  “design  parameter”  costs,  but  add  two  boundary  cases  of  significance.  First,  minor 
changes  often  do  not  require  a  reworking  of  the  module  parameters,  largely  owing  to  the 
allowances  of  play  existing  within  the  system.  Second,  major  changes  usually  require  a  much 
more  costly  reworking  of  the  module  concept  itself.  Although  they  do  not  use  the  same 
terminology  as  Baldwin  and  Clark,  the  implication  is  that  these  large  changes  could  challenge 
even  the  initial  design  rules.  Between  those  two  extremes,  however,  Holtta  and  Otto  observe  a 
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roughly  linear  relationship  between  the  degree  of  change  requested  and  the  difficulty — and,  by 
inference,  the  cost — of  enacting  that  change. 

In  terms  of  production,  Fixson  (2007),  in  his  review  of  research  into  modularity  and 
commonality,  finds  that  most  studies  of  modularity  have  identified  economies  of  scale  as  a 
significant  cost  benefit.  Garud  and  Kumaraswamy  (1995)  describe  the  effect  as  an  economy  of 
substitution.  The  ability  to  manufacture  components  separately  from  the  products  they 
comprise  permits  these  component  designs  to  outlive  individual  product  lines.  Thus,  modularity 
extends  the  size  of  the  production  runs  across  both  products  and  through  time.  This  reuse  of  a 
design  lowers  costs  by  reducing  retooling  requirements.  The  relationship  is  not  entirely  linear. 
There  is  an  optimal  number  of  modules  where  increasing  assembly  costs  balance  out  the 
decreasing  fabrication  costs  (Fixson,  2007). 

The  scale  of  the  product  itself  may  also  be  significant  in  whether  these  cost  benefits  can 
be  realized.  Zhang  and  Gershenson  (2003),  investigating  a  collection  of  fourteen  small- 
consumer  products,  “found  no  general  relationships  between  relative  modularity  and  cost,  or 
between  change  in  modularity  and  change  in  cost.” 

In  sales  and  demand  for  commercial  products,  Desai,  Kekre,  Radhakrishnan,  and 
Srinivasan  (2001)  find  that  increasing  commonality  between  products  can  hurt  demand.  Shared 
components  reduce  the  perceived  value  of  high-value  products  and  increase  the  component 
costs  for  low  value  products,  thus  eating  into  profits  from  both  ends.  The  F-35  Joint  Strike 
Fighter  offers  an  interesting  case  of  commonality  across  systems  in  a  military  context.  Its  three 
variants  are  designed  for  three  different  service  applications  (a  traditional  fighter  for  the  Air 
Force,  a  vertical/short  take-off  and  landing  vehicle  for  the  Marines,  and  a  carrier-based  fighter 
for  the  Navy).  If  successful,  this  approach  demonstrates  a  way  whereby  commonality  increases 
demand  via  appealing  to  different  classes  of  customers. 

In  sustainment,  modularity  can  help  reduce  inventory  cost  by  pooling  demands,  an 
extension  of  the  economies  of  scale  that  benefit  the  production  stage.  These  early  findings 
have  seen  much  elaboration  and  investigation,  leaving  the  inventory  phase  one  of  the  most 
researched  phases  in  the  lifecycle  of  modular  architecture.  Fixson  (2007)  offers  a  thorough 
account  of  the  various  exceptions  and  extensions  of  the  inventory  research,  including  the  roles 
of  demand  distributions,  correlated  demands,  component  cost  structures,  inventory  time 
horizon,  process  and  supply  networks,  and  other  constraints  and  considerations. 

Aside  from  inventory,  the  sustainment  phase  of  the  product  lifecycle  is  one  of  the  least 
researched  aspects  of  modularity.  Gershenson  et  al.  (1999)  speculate  that  maintenance  costs 
should  diminish  with  increased  modularity,  but  their  focus  is  elsewhere,  and  they  do  not  back 
this  speculation  with  data.  Newcomb,  Bras,  and  Rosen  (1998)  demonstrate  that  it  is  possible  to 
modularize  a  product  with  respect  to  lifecycle,  i.e.  maintenance  and  disposal.  Tsai,  Wang,  and 
Lo  (2003)  offer  a  similar  demonstration.  Both  papers  indicate  that  modularity  can  reduce  costs 
of  ownership  but  only  if  applied  properly.  Gershenson,  Prasad  &  Zhang  (2003)  speculate  that 
any  modularity  is  good  for  maintenance  costs;  however,  this  hypothesis  does  not  yet  appear  to 
have  been  tested  in  the  research  literature. 

Modularity  is  related  to  the  notion  of  open  systems,  which  have  been  adopted  as  an 
initiative  in  the  DoD  acquisition.  An  open-systems  approach  seeks  to  enable  the  integration  of 
current  and  future  capabilities  into  a  system  via  standards.  Ford  and  Dillard  (2008)  study  the 
interaction  between  evolutionary  acquisition  and  open  systems  and  find  that  the  use  of  the  two 
together  may  improve  schedule  and  cost  performance  but  may  also  increase  cost  in 
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sustainment  due  to  a  trade-off  between  increased  integration  risks  (due  to  evolving  standards) 
and  reduced  design  risks  (due  to  use  of  currently  stable  standards). 

There  are  several  hypotheses  that  are  of  interest  when  considering  modularity.  These 
include,  along  with  supporting  evidence  from  the  literature: 

1 .  Increasing  modularity  decreases  the  cost  of  implementing  technology  upgrades  for 
deployed  systems  (Fleming  &  Sorenson,  2001;  Garud  &  Kumaraswamy,  1995; 
Gershenson  et  al.,  2003;  Huang  &  Kusiak,  1998;  Ulrich  &  Tung,  1991 ;  Ulrich,  1995); 

2.  Increasing  modularity  decreases  the  mean  time  to  repair  a  system  that  has  failed 
(Cheung  &  Hausman,  1995;  Gershenson  et  al.,  2003;  Tsai  et  al.,  2003); 

3.  Increasing  modularity  increases  the  upfront  engineering  design  hours  required  for  a 
system  (Ulrich,  1995); 

4.  Increasing  modularity  increases  the  cost  of  changes  to  infrastructure  (Ethiraj  &  Levinthal, 
2004;  Fleming  &  Sorenson,  2001;  Garud  &  Kumaraswamy,  1995;  Ulrich  &  Tung,  1991; 
Ulrich,  1995). 

It  should  be  noted  that  Fleming  and  Sorenson  (2001)  offer  mixed  support  for  hypothesis 
1  since  they  find  that  small  technology  upgrades  are  handled  easily  with  a  modular  architecture 
but  that  major  upgrades  may  pose  challenges  since  they  may  require  changes  to  the  modular 
architecture  itself.  In  addition,  Garud  and  Kumaraswamy  (1995)  assert  that  technology  upgrade 
costs  decrease  only  at  the  expense  of  an  initial  infrastructure  cost.  This  paper  primarily 
addresses  the  first  two  hypotheses. 

As  the  number  of  components  in  a  system  increases,  it  is  a  complex  task  to  compare 
different  modularity  matrices  and  quantitatively  determine  differences  in  modularity.  Thus,  there 
has  been  interest  in  establishing  a  modularity  index  to  provide  a  standardized  measurement  of 
modularity.  Two  such  indices  are  given  by  Guo  and  Gershenson  (2004)  and  Holtta-Otto  and  de 
Week  (2007).  Effective  modularity  indices  remain  an  area  of  research. 

3.  Model  Description 

This  research  uses  a  simulation-based  decision  support  to  determine  the  effectiveness 
of  different  acquisition  policies  and  processes.  Simulation  has  traditionally  been  used  in 
process-based  domains  such  as  manufacturing  (Law  &  Kelton,  2000).  Increasingly,  it  is  being 
used  to  study  acquisition.  Ford  and  Dillard  (2008)  use  a  system  dynamics  approach,  which 
models  the  delayed  effects  and  feedback  flows  associated  with  the  acquisition  enterprise. 
Discrete-event  simulation  is  used  in  our  previous  work  (Pennock  &  Rouse,  2008)  and  by  Olson 
and  Sage  (2003).  Discrete-event  simulation  tends  to  offer  better  representational  support  for 
organizational  decision-making  processes. 

3.1.  Existing  Model  Summary 

Our  existing  model  is  implemented  using  ARENA  10.0,  a  commercially  available, 
discrete-event  simulation  package.  It  consists  of  three  interacting  components,  which  address 
the  traditional  acquisition  system  (Pennock  &  Rouse,  2008): 

■  Technical  Progress  Model.  The  technical  progress  model  accounts  for  basic 
research  that  occurs  exogenous  to  the  defense  enterprise.  This  work  may  be 
performed  in  the  commercial  sector  or  via  government  funding.  It  feeds  raw,  new 
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technologies  into  a  technology  development  process  model  that  reflects  the 
DoD's  science  and  technology  (S&T)  development  enterprise.  Technologies  are 
characterized  by  an  application  area,  a  maturity  level  and  a  capability  level.  An 
example  of  an  application  area  might  be  radar.  The  maturity  level  reflects  the 
readiness  of  the  technology  for  usage  and  is  measured  using  the  NASA 
technological  readiness  level  (TRL)  scale,  recently  adopted  by  the  DoD  (DoD, 
2006,  July).  Capability  level,  on  the  other  hand,  represents  the  technology's 
capability  (once  deployed)  in  relation  to  previous  generations  within  the  same 
application  area.  Capability  level  for  each  succeeding  generation  is  determined 
by  a  combination  of  a  learning  effect  (from  the  other  DoD  applications)  and  an 
exogenous  progress  effect  (from  commercial  and  outside  technical  progress). 
Technologies  are  put  into  the  technology  development  process  model  at  an  early 
TRL  (e.g.,  1). 

■  Technology  Development  Process  Model.  In  this  S&T  enterprise,  new 
technologies  for  the  DoD  systems  typically  undergo  a  staged  process  of 
development  whereby  ideas  are  reduced  to  working  technologies  that  can  be 
integrated  into  a  system.  There  is  considerable  technical  risk  in  the  development 
process,  as  ideas  often  do  not  work  in  practice,  do  not  scale-up  to  production,  or 
do  not  integrate  into  systems.  The  staged  process  mitigates  risk  by  not  fully 
funding  a  technology's  development,  allowing  it  to  be  culled  if  it  fails  or  if  it  is 
outpaced  by  competing  technologies.  It  should  be  noted  that  the  S&T  enterprise 
model  consists  of  a  single,  unified  organization  rather  than  the  myriad  agencies 
that  comprise  the  actual  DoD  S&T  enterprise. 

■  System  Acquisition  Process  Model.  The  system  acquisition  model  primarily 
represents  the  first  four  phases  of  a  defense  acquisition  program,  as  specified  in 
the  DoD  Defense  Acquisition  Guidebook  (2006).  These  include  concept 
development,  technology  development,  system  development  and  production  & 
deployment.  Operations  &  support  is  represented  by  a  simple  delay  function  for 
the  period  of  sustainment.  The  system  acquisition  process  model  pulls 
technologies  from  the  technology  development  process  model  for  use  in  the 
system  being  developed.  In  the  existing  model,  the  TRL  at  which  these 
technologies  are  selected  is  an  experimental  variable  used  to  assess  the  effect  of 
traditional  acquisition  (which  selects  relatively  immature  technologies  and 
matures  them  in  the  program  for  significant  capability  leaps  in  deployed  systems) 
versus  evolutionary  acquisition  (which  selects  relatively  mature  technologies  for 
more  frequent,  but  smaller  capability  leaps). 

The  remainder  of  this  section  discusses  two  enhancements  to  the  existing  model — the 
introduction  of  a  representation  for  system  modularity  and  a  model  of  the  sustainment  phase  of 
the  acquisition  lifecycle. 

3.2.  Modularity  Matrix 

A  system  is  assumed  to  have  n  components.  These  components  may  or  may  be  related 
with  one  another  for  the  purposes  of  repair/replacement  and/or  technology  upgrades  during 
sustainment.  One  of  these  components  is  designated  as  the  system  infrastructure,  or  the 
platform  that  integrates  the  various  components.  Modular  systems  often  require  such  an 
infrastructure  to  facilitate  modularity.  Modularity  is  then  characterized  as  the  degree  to  which 
the  various  components  interact  or  are  connected,  and  it  is  represented  as  an  n  x  n  matrix.  It 
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should  be  noted  that  modularity  is  assumed  to  be  a  function  of  the  system  design,  as 
determined  in  upstream  stages  of  the  acquisition  process. 

Each  entry  m,y  in  this  modularity  matrix  M  represents  the  probability  that  a  change  in 
component  /  necessitates  a  change  in  component  j.  Component  failures  and  component 
technology  upgrade  opportunities  arrive  and  involve  changes  to  a  component.  Due  to 
modularity  effects,  they  may  also  involve  changes  to  other  components  through  the  relations 
represented  by  M.  The  modularity  values  for  a  particular  system  may  differ  for  repairs  and 
technology  upgrades,  resulting  in  two  different  matrices,  Mrand  Mf.  Also,  a  modularity  matrix  is 
not  necessarily  symmetric.  That  is,  changes  to  component  /  may  affect  component  j  in  a 
manner  different  from  that  in  which  changes  to  j  affect  /.  A  simple  example  of  asymmetry  is 
when  replacing  /  requires  removing  j,  but  replacing  j  does  not  require  removing  /.  Components 
may  be  organized  into  modules  in  complex  systems. 

3.3.  Sustainment  Model 

The  sustainment  model  has  two  primary  processes — repairs  and  technology  upgrades. 
Failures  and  technology  upgrade  opportunities  arrive  as  random  events  to  a  deployed  system, 
according  to  a  Poisson  process  with  a  particular  rate.  Each  failure  or  technology  upgrade 
opportunity  directly  affects  only  one  component,  except  that  an  infrastructure  component,  when 
present,  is  not  affected  by  failures  or  technology  upgrades  and  is  assumed  to  be  component  1 . 
However,  repairs  or  upgrades  may  cascade  to  other  components,  due  to  modularity 
relationships.  The  following  notation  is  used  for  the  sustainment  model. 

■  fj  is  the  failure  rate  associated  with  component  i.  U  is  undefined  when  infrastructure 
is  present  (since  infrastructure  is  component  1). 

■  q  is  the  repair  rate  associated  with  component  i.  p  is  undefined  when  infrastructure 
is  present. 

■  ti  is  the  arrival  rate  of  new  technology  upgrades  for  component  i.  U  is  undefined 
when  infrastructure  is  present. 

■  Uj  is  the  upgrade  rate  for  component  /'.  u-\  is  undefined  when  infrastructure  is  present. 

■  pi  is  the  cost  of  repairing  component  /'.  pi  is  undefined  when  infrastructure  is  present. 

■  q i  is  the  cost  associated  with  a  technology  upgrade  to  component  i.  q-\  is  undefined 
when  infrastructure  is  present. 

■  C/j  is  the  compatibility  cost  associated  with  making  component  j  technologically 
compatible  with  component  /  if  /  is  upgraded  and  if  the  interaction  between  /  and  j 
necessitates  that  j  be  made  compatible  to  the  new  technology  for  i.  cn  is  undefined 
when  infrastructure  is  present. 

In  general,  it  is  assumed  that  fj  >  f„  q  >  uh  and  p,  <  q,. 

The  simulation  logic  works  as  follows.  When  a  failure  to  component  /  arrives  to  the 
system,  it  invokes  a  repair  delay  for  that  component,  occurring  at  rate  q.  All  components  j  such 
that  rriij  >  0  are  evaluated  probabilistically  via  a  Bernoulli  variable,  using  the  probability  m(J,  to 
determine  whether  j  must  also  be  repaired.  Any  components  j  requiring  a  repair  are  then 
repaired  at  rate  q.  This  repair  requirement  can  cascade  to  additional  components  that  are 
dependent  on  j,  and  so  on.  The  system  experiences  a  repair  downtime  equal  to  the  maximum 
repair  time  of  /  and  that  of  any  other  affected  components. 
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Similarly,  when  a  technology  upgrade  to  component  /  arrives  to  the  system,  it  invokes  an 
upgrade  delay  for  that  component.  This  delay  occurs  at  rate  u,.  All  components  j  such  that  m,y  > 
0  are  evaluated  probabilistically  via  a  Bernoulli  variable  to  determine  whether  j  must  also  be 
made  compatible  with  the  upgrade.  Any  components  j  requiring  a  compatibility  operation  invoke 
a  delay  at  rate  Uj.  Upgrade  effects  can  cascade  similarly  to  repair  effects.  The  system 
experiences  an  upgrade  downtime  equal  to  the  sum  upgrade  time  of  /  and  compatibility  time  of 
any  affected  components.  This  is  in  contrast  to  the  downtime  due  to  repairs. 

If  a  failure  or  technology  upgrade  for  /  arrives  while  the  system  is  in  downtime,  then  that 
failure  or  technology  upgrade  queues  until  the  downtime  is  resolved.  Multiple  entities  in  this 
queue  are  processed  as  first-come-first-served. 

Clearly,  this  is  a  relatively  simple  model.  It  is  meant  to  allow  basic  analysis  of  the  effects 
of  modularity  and  to  provide  a  basis  for  more  complex  models  in  the  future. 

4.  Experiment 

In  this  section,  we  detail  a  simulation  experiment  to  test  the  effect  of  different  types  of 
modularity  matrices  on  sustainment.  The  dependent  variables  are  the  repair  costs,  upgrade 
costs  and  system  availability.  Sustainment  of  a  single  system  is  considered  in  each 
experimental  run.  Three  classes  of  modularity  are  considered: 

■  Type  1 .  All  non-diagonal  entries  in  the  matrix  are  the  same  fractional  probability 
value. 

■  Type  2.  All  non-diagonal  entries  in  the  modularity  matrix  are  either  zero  or  one. 

■  Type  3.  The  matrix  consists  of  modules,  comprised  of  components  that  have  strong 
relationships,  but  the  relationship  entries  between  modules  in  different  components 
is  zero. 

4.1.  Parameters  and  Assumptions 

The  simulation  is  executed  over  a  period  representing  ten  years  of  sustainment.  The 
following  parameter  values  are  used.  These  parameter  values  are  selected  as  notional  values 
for  the  experimental  analysis  to  illustrate  the  effects  of  the  modularity. 

■  fj  =  60  days  for  all  / 

■  r,=  1  hour  for  all  / 

■  ti  =  360  days  for  all  / 

■  Uj  =  6  hours  for  all  / 

■  p,  =  10  currency  units  for  all  / 

■  g,  =  100  currency  units  for  all  / 

■  Cj=  15  currency  units  for  all  /  and  j 

4.2.  Experimental  Setup 

Table  1  shows  the  variations  tested  among  the  different  types  of  modularity  matrices.  In 
matrices  of  types  1  and  2,  n  equals  10.  In  matrices  of  type  3,  the  size  is  adjusted  to  n  equals  16 
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to  accommodate  modules  being  the  same  size  (e.g.,  systems  with  eight  modules,  each  having 
two  components,  or  with  four  modules,  each  having  four  components). 


Table  1.  Modularity  Matrix  Variations  Tested 


Matrix  Type 

Variations 

Type  1 

Eleven  different  variations  are  simulated.  Each  variation  uses  a  different  value 
for  all  non-diagonal  m-j.  The  different  values  used  are  0.0,  0.1 , 0.2,  0.3,  0.4, 

0.5,  0.6,  0.7,  0.8,  0.9,  and  1.0. 

Type  2 

Seven  different  variations  are  simulated.  Each  variation  has  a  mix  of  values 
(0,  1)  for  non-diagonal  m,y.  Each  variation  uses  a  different  probability  to  select 
a  specific  value  for  each  m,y.  These  probabilities  are  0.0,  0.1,  0.2,  0.3,  0.4,  0.5, 
and  0.6,  and  the  probability  corresponds  to  m,y  equaling  one,  as  opposed  to 
zero.  It  was  determined  that  probability  values  above  0.6  had  similar  behavior; 
thus,  they  are  not  considered  here. 

Type  3 

Five  variations  are  simulated.  Each  variation  has  a  different  size  of  module. 

The  variations  include  sixteen  modules  of  size  1 ,  eight  modules  of  size  2,  four 
modules  of  size  4,  two  modules  of  size  8,  one  module  of  size  16.  Within  each 
module,  all  m,y  equal  1.  Relationships  between  modules  have  my  equal  0. 

Ten  replications  of  each  variation  are  run  for  statistical  significance. 

5.  Results  and  Analysis 

5.1.  Repair  Costs 

Figures  2-4  illustrate  average  repair  costs  as  a  function  of  the  level  of  modularity  in  a 
system.  The  actual  average  repair  cost  shown  is  the  average  collateral  repair  cost,  or  the  cost 
of  repairing  other  components  related  to  a  failed  component  that  must  be  repaired  due  to  a 
modularity  relationship.  This  shows  the  variable  effect  of  modularity  in  terms  of  average  repair 
cost.  The  result  from  each  replication  across  each  variation  is  shown  in  each  figure.  The  units 
for  cost  are  in  currency  units,  as  specified  in  the  parameters  for  the  model. 

According  to  expectations,  as  the  level  of  relationship  strength  (or  coupling)  increases 
(i.e.,  as  modularity  decreases),  the  repair  cost  increases  for  each  type  of  matrix.  The  factors  of 
interest  include  the  points  at  which  the  costs  start  to  converge  to  a  maximum  value  and  the 
relative  spread  of  the  costs  for  each  level  of  variation  within  each  type  of  modularity  matrix.  In 
the  type  1  matrix,  the  variance  is  less  than  that  of  the  type  2  matrix,  suggesting  that  numerous 
weak  relationships  provide  a  more  predictable  repair  cost  for  a  system  than  a  set  of 
relationships  that  are  either  very  strong  or  very  weak.  Intuitively,  this  makes  sense.  It  also  is 
reinforced  by  the  outcome  from  type  3  matrices,  in  which  the  repair  cost  is  always  the  same, 
since  a  component  failure  leads  to  replacement  of  the  entire  module,  and  each  module  is  the 
same  size  and  cost.  Since  module  size  has  a  linear  relationship  with  module  cost,  the  cost 
relationship  with  modularity  is  likewise  linear. 

Since  the  failure  rates  are  the  same  across  all  replications,  the  patterns  for  total  repair 
costs  of  each  replication  (over  the  entire  ten-year  time  horizon)  are  similar  to  those  of  average 
costs  (per  failure  incident).  Therefore,  only  the  average  costs  are  shown.  However,  it  should  be 
noted  that  there  would  be  variance  across  variations  in  the  type  3  matrix  total  costs  since  the 
number  of  failures  during  the  time  horizon  is  a  random  variable. 
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Average  Collateral  Repair  Cost 


Coupling  (value  of  mij  in  type  1  modularity  matrix) 


Figure  2.  Repair  Cost  as  a  Function  of  Modularity  for  Type  1  Matrix 


Average  Collateral  Repair  Cost 


Coupling  (Probability  that  mij  =  1  in  type  2  modularity  matrix) 


Figure  3.  Repair  Cost  as  a  Function  of  Modularity  for  Type  2  Matrix 
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Average  Collateral  Repair  Cost 


Size  of  Modules  (number  of  coupled  components  in  each  module  in  type  3  matrix) 


Figure  4.  Repair  Cost  as  a  Function  of  Modularity  for  Type  3  Matrix 


5.2.  Technology  Upgrade  Costs 

Figures  5-7  illustrate  average  upgrade  costs  as  a  function  of  the  level  of  modularity  in  a 
system.  Average  upgrade  cost  addresses  the  work  to  make  components  consistent  to 
upgrades  when  they  are  related  to  the  component  being  upgraded,  i.e.,  the  variable  portion  of 
cost  related  to  modularity.  The  result  from  each  replication  across  each  variation  is  shown  in 
each  figure. 

The  behavior  patterns  for  upgrade  costs  are  comparable  to  those  for  repair  costs:  as  the 
level  of  relationship  strength  increases,  the  upgrade  cost  increases  for  each  type  of  matrix.  As 
with  repair  costs,  the  pattern  for  total  upgrade  cost  over  the  ten-year  time  horizon  is  similar  to 
that  of  the  average  cost,  so  only  the  average  costs  are  shown.  The  units  for  cost  are  in 
currency  units,  as  specified  in  the  parameters  for  the  model. 
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Average  Collateral  Upgrade  Cost 


Coupling  (value  of  mij  in  type  1  modularity  matrix) 


Figure  5.  Upgrade  Cost  as  a  Function  of  Modularity  for  Type  1  Matrix 


Average  Collateral  Upgrade  Cost 


Coupling  (Probability  that  mij  =  1  in  type  2  modularity  matrix) 

Figure  6.  Upgrade  Cost  as  a  Function  of  Modularity  for  Type  2  Matrix 
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Average  Collateral  Upgrade  Cost 


Size  of  Modules  (number  of  coupled  components  in  each  module  in  type  3  matrix) 


Figure  7.  Upgrade  Cost  as  a  Function  of  Modularity  for  Type  3  Matrix 
5.3.  System  Downtime 

Finally,  Figures  8-10  illustrate  average  system  downtime  during  the  ten-year  time 
horizon  as  a  function  of  the  level  of  modularity  in  a  system.  Average  downtime  is  a  combined 
effect  of  failures  and  technology  upgrades.  The  result  from  each  replication  across  each 
variation  is  shown  in  each  figure. 

As  the  level  of  relationship  strength  increases,  the  average  downtime  increases  for  each 
type  of  matrix.  The  behavior  patterns  are  somewhat  similar  to  those  for  costs.  The  average 
downtime  values  across  matrix  type  3  variations  are  not  constant,  due  to  the  random  number  of 
failures  and  technology  upgrades  in  each  replication.  The  units  for  downtime  are  the  fraction  of 
time  that  the  system  is  unavailable. 
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Average  Downtime  Average  Downtime 


Average  Downtime 


Coupling  (value  of  mij  in  type  1  modularity  matrix) 


Figure  8.  Downtime  as  a  Function  of  Modularity  for  Type  1  Matrix 


Average  Downtime 


Coupling  (Probability  that  mij  =  1  in  type  2  modularity  matrix) 


Figure  9.  Downtime  as  a  Function  of  Modularity  for  Type  2  Matrix 
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Average  Downtime 


Size  of  Modules  (number  of  coupled  components  in  each  module  in  type  3  matrix) 


Figure  10.  Downtime  as  a  Function  of  Modularity  for  Type  3  Matrix 


6.  Discussion  and  Future  Research 

These  results  provide  some  insight  into  the  effect  of  modularity  on  sustainment  costs 
and  system  availability.  There  is  some  potential  for  cost  savings  and  improved  system 
availability  as  modularity  is  increased.  Clearly,  the  parameter  values  and  complexities  of  real 
systems  need  to  be  considered,  and  this  will  be  a  focus  of  future  research  efforts.  Such  efforts 
need  to  account  for  the  notion  of  integration  risk  over  the  lifecycle,  as  detailed  in  Ford  &  Dillard 
(2008). 


One  major  goal  of  this  research  is  to  characterize  the  effect  of  modularity  over  the 
acquisition  lifecycle.  Thus,  current  work  is  focusing  on  integration  of  the  existing  model  of 
acquisition  with  the  new  sustainment  and  modularity  models.  This  involves  modeling  modularity 
and  its  engineering  costs  in  the  acquisition  model  as  well  as  modeling  the  flow  of  technology 
upgrades  to  the  sustainment  model  from  the  S&T  model.  The  emphasis  on  cost  modeling  will 
be  on  parametric  models  for  cost  estimation  (e.g.,  Valerdi  &  Liu,  in  press).  Such  models  must 
address  not  only  the  initial  design  of  modularity  but  also  adjustments  during  development  such 
as  evolution  of  design  parameters  (Baldwin  &  Clark,  2000).  The  hypothesis  is  that  modularity 
tends  to  increase  design  and  development  costs  while  decreasing  production  and  sustainment 
costs.  The  question  is  to  determine  what  levels  of  modularity,  in  combination  with  other  system 
characteristics,  achieve  the  best  results,  not  only  in  terms  of  cost  but  also  in  terms  of  time  to 
deployment  and  post-deployment  availability.  One  such  system  characteristic  is  production 
level,  which  has  the  potential  to  leverage  economies  of  scale  in  making  modularity  more  cost 
effective. 

To  answer  the  question  about  the  effectiveness  of  modularity  levels,  it  is  important  to  be 
able  to  characterizer  modularity  by  a  standard  metric  such  as  a  modularity  index.  This  also  will 
be  an  avenue  of  future  research. 
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Another  goal  is  to  study  the  effect  of  enterprise  characteristics  and  their  interactions  with 
system  characteristics.  In  particular,  we  are  interested  in  studying  the  effects  of  alignment  in  the 
S&T  system  and  the  concept  of  mission  risk.  The  current  model  assumes  a  unitary  S&T 
organization  rather  than  the  multi-organization  S&T  enterprise.  In  terms  of  cost,  schedule  and 
risk,  what  is  the  trade-off  between  the  redundancy  of  a  multi-organization  S&T  enterprise  versus 
the  efficiency  of  a  unitary  organization?  Mission  risk  is  increasingly  important,  given  the 
evolution  of  threats  that  need  to  be  addressed.  Does  modularity  aid  in  adapting  systems  in  the 
field  to  new  mission  requirements?  Finally,  we  plan  to  extend  previous  results  by  exploring 
which  conditions  from  the  above  areas  of  study  make  evolutionary  acquisition  more  favorable 
than  traditional  approaches. 
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Motivation 
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•  Previous  findings  indicate: 

-  Evolutionary  acquisition  can  result  in  faster  deployment  of 
capability 

-  But  may  result  in  increased  overhead  cost  due  to  more  frequent 
acquisition  cycles 

•  In  general,  what  factors  cause  evolutionary  acquisition  to 
be  more  effective  than  traditional  acquisition: 

-  Lifecycle  cost 

-  Timeliness  of  deployed  capability 

-  Availability  of  new  systems  in  the  field 

•  In  particular,  what  role  does  system  modularity  play: 

-  Lifecycle  cost 

-  System  availability 
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Existing  Model 
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Model  Summary 
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•  Technical  progress  model 

-  Addresses  research  exogenous  to  acquisition  enterprise 

-  Results  are  input  to  S&T  model 

•  S&T  model 

-  Addresses  maturation  of  technologies  via  a  staged  process 

-  Incorporates  technical  risk 

-  Assumes  single  S&T  organization 

•  Acquisition  model 

-  Primarily  addresses  concept  development,  technology 
development,  system  development  and  production  &  deployment 

-  Pulls  technologies  from  S&T  model 
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Modularity 


•  Independence  of  different 
system  components 

•  Common  infrastructure  and 
standard  interfaces 

•  Major  principle  in  product  and 
system  design  literature 

-  Increased  modularity 
decreases  cost/time  for 
repairs  and  technology 
upgrades  in  sustainment 

-  Increased  modularity 
increases  cost  of  design 

-  Increased  modularity  may 
increase  costs  for  changes  to 
infrastructure 
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Modularity  Model 
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Systems  consist  of  components  or 
modules  (i.e.,  collection  of 
components) 

A  relationship  between 
components  i  and  j  exists  if 
changes  to  i  causes  changes  to  j 

Assume  this  relationship  is 
characterized  by  a  probability  that 
a  change  to  i  causes  a  change  to  j 

Modularity  can  be  represented  as 
a  matrix 

This  matrix  is  not  necessarily 
symmetric 

Diagonal  elements  are  not 
relevant 
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Modularity  Examples  Ge^isr™ 
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Sustainment  Model 
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Model  Summary 
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•  Addresses  repairs  and  technology  upgrades  for  systems 
in  the  field 

•  Each  failure  or  technology  upgrade  affects  only  one 
component 

•  But  due  to  relationships,  failures  and  technology 
upgrades  can  affect  other  components 

•  Failures  and  technology  upgrades  are  assumed  to  occur 
via  a  Poisson  process 
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Sustainment  Parameters  G^ssr“ 


•  fj  is  the  failure  rate  associated  with  component  i,  fi  is  undefined  when 
infrastructure  is  present  (since  infrastructure  is  component  1), 

•  n  is  the  repair  rate  associated  with  component  i.  n  is  undefined  when 
infrastructure  is  present. 

•  tj  is  the  arrival  rate  of  new  technology  upgrades  for  component  i.  ti  is 
undefined  when  infrastructure  is  present. 

•  Uj  is  the  upgrade  rate  for  component  i.  ui  is  undefined  when  infrastructure  is 
present. 

•  pi  is  the  cost  of  repairing  component  i.  pi  is  undefined  when  infrastructure  is 
present. 

•  qi  is  the  cost  associated  with  a  technology  upgrade  to  component  i.  qi  is 
undefined  when  infrastructure  is  present. 

•  Gy  is  the  compatibility  cost  associated  with  making  component  j 
technologically  compatible  with  component  i  if  i  is  upgraded,  and  if  the 
interaction  between  i  and  j  necessitates  that  j  be  made  compatible  to  the 
new  technology  for  i.  cn  is  undefined  when  infrastructure  is  present. 
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Parameter  Values 
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•  Matrix  has  10  components 

-  Adjusted  to  16  for  systems  with  modules 

•  fi  =  60  days  for  all  i 

•  n  =  1  hour  for  all  i 

•  ti  =  360  days  for  all  i 

•  Uj  =  6  hours  for  all  i 

•  pi  =  10  currency  units  for  all  i 

•  qi  =  100  currency  units  for  all  i 

•  Cij  =  1 5  currency  units  for  all  i  and  j 
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Experiment 
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•  Independent  variable  -  relationship  values  within  a  class 
of  modularity  matrix  types 

-  Relationship  Strength  (Type  1)  -  All  non-diagonal  matrix 
elements  have  the  same  probability  value  (this  value  ranges 
from  0  to  1 ) 

-  Relationship  Number  (Type  2)  -  All  non-diagonal  matrix  entries 
are  either  0  or  1  (number  of  1’s  determined  randomly  by 
probability  ranging  from  0  to  0.6) 

-  Modules  (Type  3)  -  Matrix  is  composed  of  modules  of  varying 
size  (number  of  modules  ranges  from  1  to  16) 

•  Dependent  variables  -  repair  costs,  upgrade  costs  and 
system  availability 

•  Time  horizon  -  10  years  of  system  operation 
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Independent  Variables  Ge^|»" 
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Results  Summary 
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•  Major  cost  benefits  for  high  levels  of  modularity,  with 
diminishing  returns  as  modularity  decreases 

•  Systems  with  varied  number  of  strong  relationships 
exhibit  greater  cost  variability  than  those  with  varied 
strength  of  relationships 

•  Systems  with  modules  (as  opposed  to  components) 
exhibit  a  linear  cost  effect  (increasing  cost  as  module 
size  increases) 

•  Availability  exhibit  similar  behavior  (with  more  variability) 
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Repair  Cost  —  Strength  Ge^|»" 
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Repair  Cost  —  Number  Ge^h|s"“ 
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Repair  Cost  —  Modules 
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Upgrade  Cost  —  Strength  Ge^4s=?™ 
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Upgrade  Cost  —  Number  “■wpaass™ 
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Upgrade  Cost  —  Modules  Ge^4s=?™ 


Average  Collateral  Upgrade  Cost 
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Average  Downtime 


Availability  —  Strength  Ge^|»" 
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Average  Downtime 


Availability  —  Number  Ge^|»" 
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Average  Downtime 


Availability  -  Modules  Ge^|»" 
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Conclusions 
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•  Models  developed  to  study  effect  of  system  modularity  in 
sustainment 

•  Simulation  experiments  demonstrated  effects  of  different 
patterns  of  modularity  in  terms  of 

-  Repair  costs 

-  Technology  upgrade  costs 

-  System  availability 


Knowledge  and  Skills  for  Enterprise  Transformation. 


25 


Future  Research  1 


•  Develop  a  model  of  engineering  costs  for  design  and  development 
and  production  of  modularity  in  systems 

-  Study  trade-offs  between  design/development  and  sustainment  costs 
and  availability 

•  Characterize  modularity  via  a  standardized  modularity  index 

-  Aid  in  categorization  and  experimentation 

•  Integrate  sustainment  model  with  existing  acquisition  model  to 
support  analysis  of  effectiveness  of  evolutionary  acquisition  with 
regard  to 

-  Mission  risk 

-  S&T  alignment  and  funding  strategy 

•  Analyze  real  systems  with  this  framework 

-  UAS  and  JSF 
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Future  Research  2 


•  Move  beyond  process-oriented  representations  to 
incorporate  organizational  behavior 

-  Human  behavior  via  character  models 

-  Social  and  organizational  networks 

-  Eco-system 

-  Organizational  stories  via  drama  management 

•  Use  organizational  simulation  to  study  role  of  incentives 
and  information  in  acquisition  enterprise  performance 
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